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(54) Improvements in or relating to the prediction of product defects 



(57) A software predictive engine is described that 
provides a tool to analyze the import of defection the 
software development process. By providing a predic- 
tion of escaping defects, in terms of the normalized sig- 
ma measure and the predicted lines of code, together 



with historical data the impact of defects at the stages 
of the design can be predicted. As actual defects are 
measured predicted sigma level can also be predicted. 
The engine includes generating a defect containment 
matrix for the historical data and for the current software 
program being developed. 
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Description 

TECHNICAL FIELD OF THE INVENTION 

5 This invention is related to a method and apparatus for predicting product defects. 

BACKGROUND OF THE INVENTION 

Texas Instruments Incorporated Defense Systems and Electronics Group (DSEG), a 1992 Malcolm Baldrige Qual- 
10 ity Award Winner, lias developed both the technology and deployment infrastructure for a quality initiative named Six 
Sigma. Aimed at measuring and reducing variation in a product or sen/ice, this technology has allowed many Texas 
Instruments disciplines, including software development, to quantiflably progress towards zero defects. Six Sigma 
originated with a concept that variation in a product or service is the culprit behind defects. Consequently, one can 
begin with customer requirements and identify units of product or service in which opportunities for defects and actual 

'S defects are itemized. Through simple computations, a sigma level of performance may be identified Higher sigma 
values represent higher quality levels. Six Sigma performance is equivalent to no more than 3.4 defects per million 
opportunities. But why do we care about Six Sigma? In terms of cost of quality a Four Sigma organization cannot 
compete with a Six Sigma organization A Four Sigma organization will spend as much as 25% of each sales dollar on 
cost of non-conformance while a Six Sigma organization will spend less than 1%. Organizations performing at Four 

20 Sigma will spend 25 times more dollars on rework, etc. than a Six Sigma organization will. Additionally Six Sigma 
analysis supports defect causal analysis, prevention and prediction. It has been incorporated into our process improve- 
ment program and is part of a widespread training program for all employees of Texas Instruments DSEG Over the 
past 5 years, we believe that this technology contributed to our lOX improvement in software fault density Six Sigma 
measurement begins with the customers and their requirements. Defect categories may then be identified along with 

25 units of product or service. From this framework, defects per unit, opportunities for defects, defects per million oppor- 
tunities and Sigma levels may be computed. Defects per million opportunities (dpmo) may be computed by simply 
dividing the defects per unit (DPU) by the opportunity count (source lines of delivered software) and then multiplvina 
by 1 million. ^ ' ^ 

The number of defects per million opportunities (dpmo) may be converted into a Sigma level via use of a conversion 
chart as shown in Figure 1. Note that changes in the dpmo levels are not linearly proportional to the Sigma levels 
Thus, for an organization of higher levels of sigma performance, smaller amounts of dpmo improvement are needed 
to achieve the same sigma improvement as that of an organization at a lower level of sigma performance Siqma 
measures support benchmarking of quality levels. 

In the development of software there are generally at least five stages - the first being the requirement stage (RA) 
the second being the preliminary design stage (PD); the third being the design detail stage (DD); the fourth the coding 
stage (CUT), and the fifth the integrating and testing stage (l&T) Defects in software development activity represent 
rework, resulting in increased cost and cycle times . 

While it is known to make estimates of escaping defects mode during a latter phase of integration and test using 
Weibull curve fittings, there is no known solution for long range prediction of the number of escaping defects 

Accordingly, a technical problem to be solved relates to the provision of methods and apparatus which can auto- 
matically predict counts of defects as part of a development process. 

SUMMARY OF THE INVENTION 

In accordance with one preferred embodiment of a present invention, there is provided a method and statistical 
tool apparatus for predicting defects in products. The method according to one embodiment includes the step of pro- 
viding historical data of defects at different stages of development and a value representing a goal for escaping defects 
Also provided is the planned total number of opportunities for defects. The goal for the number of escaping defects 
and planed number of opportunities for defects are backsolved to determine the total number of defects The total 
defects are distributed as a function of the historical data to provide prediction of defects at the different stages of 
aeveiopment. ^ 

The apparatus according to one preferred embodiment of the present invention the tool comprises a processor a 
memory and a keyboard and a drive for loading a software package and a display. The processor is loaded with a 
program for storing historical data indicating the historical pattem of defect containment in the stages of development 
The processor has stored therein algorithms for computing sigma values based on opportunities and escaping defects 
in the stages and including an algorithm for backsolving from historical data. 
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DESCRIPTION OF THE DRAWINGS 

Embodiments of the invention are described, by way of example only, with reference to the accompanying drawings, 
in which: 

5 

Fig. 1 is a conversion chart of DPMO to Sigma: 

Fig. 2 is a diagram of the personal computer used for the engine according to one embodiment of the present 
invention: 

10 

Fig. 3 is a system diagram according to one embodiment of the present invention: 
Fig. 4 is a flow chart of the operation according to one embodiment of the present invention: 
IS Fig. 5 illustrates defect flow analysis: 

Fig. 6 illustrates the historical sheet; 
Fig. 7 illustrates the current sheet: 

20 

Fig. 8 illustrates a summary sheet: and 

Fig. 9 illustrates a current sheet with equations embedded therein. 

25 DESCRIPTION OF PREFERRED EMBODIMENT 

A Software Sigma process of an embodiment of the present invention stresses defect prevention, monitoring of 
defect levels, causal analysis for process improvements and the sigma metric to [predict, early in the development cycle, 
the level of escaping defect delivered with the product. A unique aspect of the statistical model we construct to predict 

30 escaping defects is the use of project defect phase containment data in a dual manner As historical data, phase 
containment data, converted to conditional probabilities, is used to model the current state of a projects software proc- 
ess's effectiveness to prevent and capture defects. During the development phase, actual phase containment data is 
captured and utilized to make the prediction of escaping defects from the finished product. 

Variation in performance of a quality attribute causes defects. Performance of a quality attribute is often assumed 

35 to follow a normal distribution (bell shape curve) which is centered on a given mean (average) value. The potential for 
defects arises when the tails of the performance curve fall outside given specification limits. Adapted to software, there 
is no assumption made about distribution normality. In fact, distributions of interest appear to follow Poisson or Weibull 
distributions. 

In accordance with one embodiment of the present invention, a software sigma predictive engine is provided for 

40 predicting defects in software. The engine comprises, for example, a personal computer (PC) with a processor 11, a 
storage unit 13 (such as a disk drive), a monitor 15, and input device 17 such as a keyboard and mouse and 17a as 
shown in Fig. 2. The processor is loaded with a software spreadsheet program such as Microsoft Excel 5.0. The 
operation is implemented as a workbook in three sheets. The user enters data in two sheets^ the historical sheet 21 
and the current sheet 23. See diagram in Fig. 3. A third sheet 27, the summary sheet, automatically summarizes the 

-^5 decisions made, actual observed defects, the predicted number of escaping defects, and statistical confidence intervals 
qualifying the "goodness" of the prediction. There is also a fourth sheet 29 or page which contains the original computer 
program to calculate the Poisson probabilities. 

Aspects of one embodiment of the present invention are illustrated by the flow chart of Fig. 4. where historical 
defect containment data is entered (step A) in the cells of Excel 5.0 as a workbook in the historical sheet as a stage 

50 defect containment matrix. The user enters historical defect containment data for the various stages of design in a 
defect containment matrix at cell locations. This is arrived at from previous designs and detections of defects at these 
stages. Based on estimated efficiency of detection of defects, the inspection efficiencies are calculated (step B) and 
copied to the second sheet or page called the current sheet 27. The estimated overall efficiency of the process to 
detect DPU of each type over all stages is entered. The number of software lines of code (SLOC) is entered on the 

55 current workbook sheet 25 (step C) and the goal sigma value is entered on the current workbook sheet 25 (step D). 
Each SLOC represents a defect opportunity. Based on the goal sigma values and lines of code (SLOC) entered the 
total number of observed defects are calculated. The initial distribution of these defects is calculated using probability 
Pj] (step E) of a defect in ith step of being detected and removed in jth step. The upper bounds for initial defect spread 
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is calculated (step F) and the lower bounds for Initial defect spread is calculated (step G). The defect spread is copied 
down to the actual locations in the current sheet (step H) . The confidence interval is then calculated (step I). The 
statistical model used views the number of defects generated in each stage as Poisson random variables which are 
independent between the different stages of the development process. These random numbers of defects are then 
5 subjected to the random process of defect detection and the resulting number of detected, and undetected, defects 
are random variables whose distributions remain Poisson. The total number of detected defects, as well as the total 
number of escaping defects, then can be represented as the sum of independent Poisson random variables and it is 
this that makes it possible to develop an unbiased estimator for the mean number of escaping defects. An estimate of 
the variance of the estimator, along with the asymptotic normality of its distribution, allows for the calculation of an 
^0 approximate statistical confidence interval for the mean number of escaping defects, as well as an approximate pre- 
diction interval for the actual number of escaping defects. These estimates and intervals are automatically calculated 
and displayed in the spreadsheet, along with corresponding estimates and intervals for the process sigma level. The 
initial prediction is up-dated (step J) with actual data until all actuals have been entered (step K). As the design progress- 
es and actual defects are measured the actual defects are compared to the error bounds to determine if there is 
^5 consistency between the data and the model and to see if the sigma goal will be reached. 

Referring to Figure 5, there is illustrated the defect flow analysis for a software design project which starts with RA 
as a requirement and proceeds to the second Step PD, which is preliminary design, which proceeds to a detailed 
design (DD) which then proceeds to actual generation of code (CUT) and then to the fifth step of integration and test 
(l&T). Fig. 5 indicates the defect data flow model used to estimate the number of escaping defects for a typical software 
product- Some of the final defects from the requirements (RA) escape through the preliminary design, the detailed 
design, the coding and integration and test. Likewise some uncorrected defects in the preliminary design (second line) 
escape and pass on to detailed design, through the coding, and through integration and test. Some defects at the 
detailed design escape through coding and integration and test. Some coding errors escape through integration and 
test. Some integration and test errors escape. The total escaped defects per million opportunities is the defects per 
25 million opportunities (dpmo) which is converted to the Sigma level. 

The historical sheet shown in Fig. 6 is the primary location for loading historical data to prime the pump in prepa- 
ration for the collection of actual defect data during the project. The primary software sigma measure is defects per 
unit (DPU). When we associate opportunities and a chance for non-conformance, we are providing a measure of 
complexity that allows us to compare products of different size. The use of opportunities as a common denominator 
30 to compute defects per million opportunities (dpmo) is a way to normalize the different sizes between both intermediate 
and final product. Enter in cells C4:G8 of Excel 5.0 historical workbook sheet the defects for each stage observed from 
historical data or best engineering judgement. This data represents the pattern of defects detected across the project 
phases. For example the defects originated at requirements stage RA and detected at RA is 30. The defects detected 
at Integration and Test originated 1 and RA, 1 at PD, 5 at DD, 17 at CUT and 3 at l&T It represents the pattern most 
likely to occur and is transformed into conditional probabilities in cells C15:G1 9 labeled "inspection efficiencies". These 
probabilities are automatically calculated in cells C15;G19 of the historical sheet and are copied onto the current and 
summary sheets. Represented as percentages, these are the conditional probabilities that a defect generated in a prior 
or same stage will be observed in the current stage. For example, the total RA defects is 64 and the defect at C4 is 
30 so the percentage is 30/64 X 100% = 47. These percentages are then derated by the estimated efficiencies. The 
percentages listed represent de-rating of perfect capture which the conditional probabilities would suggest. In fact, the 
conditional probabilities are adjusted by the estimated efficiency percentages to indicate that our observations are not 
100% efficient. These multipliers may be estimated for programs with complete data, but without that one must use 
best engineering judgement. The spreadsheet shown in Fig. 6 has these values preset at 95% for requirements DPU, 
94% for preliminary design defects. 93% for detailed design defects, 92% for code and unit test and 90% for integration 
and test. Screening effectiveness percentages are conditional probabilities estimated from historical stage containment 
data and may be revised. 

The initial values in the current sheet (Fig. 7) allow the developer to model his environment and the effect in terms 
of DPU for given values of screen effectiveness and assumed sigma for the product. As actual data is entered for each 
phase, it overrides the DPU computed by the spreadsheet, which were needed to obtain the assumed goal sigma. 
From this point in time forward, to improve the product's sigma it is necessary to reduce the DPU observed at each 
stage by improved software process steps. The screen detection probabilities may be increased by improving the 
screens ability to detect both in-stage and out-of-stage DPU. The process is improved by finding most DPU in early 
stages of development. 

The product name, delivered SLOC (Software Lines of Code) and goal sigma are entered at E4 and E5 in the 
shaded blocks at the top of the current sheet illustrated in Fig. 7. Note that the goal sigma may be based upon Fault 
Density Data of similar products or be the development goal the project expects to attain. 

The DPU count by percentage in the historical sheet based on defects observed are computed and copied to in 
the current sheet. These percentages are used to spread the DPU in the following cells with planning DPU spread by 
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stage. The percentages may be adjusted to reflect engineering judgement if required. 

Using the percent spread, knowledge of the goal sigma of E5, the delivered SLOC E4 and the probabilities of 
detection, the engine back calculates the DPU needed in each phase to reach the sigma goal. These DPU are copied 
to the cells (C51. G55) called "defects observed" where actuals will overwrite these estimates as they are received. 

5 Note that the number of DPU passed to the next phase and the number generated (greater than observed) are also 
indicated below these cells. First the total DPU are calculated from the sigma value and SLOC and then based on 
percentage in C 1 0:G1 4. The initial values in C51 :G55 are entered. This predictive framework is based on assumptions. 

Assume a process has N steps or stages. In step i, Lj defects are introduced. Assume L^ has a Poisson distribution 
with mean and that Lj and Lj are independent for '\^]. Assume also that a defect introduced in the ith step has a 

10 probability Py of being detected and removed in the jth step, j>i, and that the detection of defects forms a Bernouli 
process and the detection of one defect is independent of the detection of other defects. It can be shown that the 
number of defects detected in stage, Kjj, has a Poisson distribution with mean XjPy and that the number of defects 
escaping from the stage they were introduced, L^-Kji. has a Poisson distribution with mean Aj(1 -Pjj). Furthermore, it can 
be shown that these random variables are independent. 

'5 This leads to the fact that the numbers of defects of a specific origin, found in each stage, have Poisson distributions 

and are all Independent of each other. Since the number of defects which escape from step i to step i+1 , Lj-Kjj, has a 
Poisson distribution with mean /.j(l-Pjj), and since these defects each have a probability Pj;^, of being detected, the 
number of stage i defects detected in step Kji^^. has a Poisson distribution with mean (1-Pji)Pji+iAj. In general, the 
number of step i defects detected (and removed) in the jth step is given by Kjj and has a Poisson distribution with mean 

20 (1-Pjj) (l-Pjj+i) -(l-Pij-i)PijAj- Furthermore^ these K^j are all independent. In addition, since they are simply the sum of 
independent Poisson variables, the total number of defects detected in each stage, the total number of detected defects 
introduced at each stage, and the total number of escaping defects all have Poisson distributions. Let 
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be the number of step t defects which have not been detected after the Nth step. By induction, it can be shown that 
Lj* has Poisson distribution. The mean of Lj* is given by 

=A-.nri - p,j) = - pjri - p^.o. . . i^i - p.^v) = a, 



where 



Let 



N 

be the total number of escaping defects. Since the L^* are independent Poisson variables, L* has a Poisson distribution 
with mean 



N N /V N 
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ESTIMATORS 

As stated before, the Kjj are independent, Poisson variables with nneans 

5 



(.-^,K.-/,,.,)...0-/-.,)PA=^T^-^ 



70 

where 



15 

This nneans that 

y 

the total number of defects from stage i which were detected, has a Poisson distribution with mean and variance given 

by: 

25 
30 

and 



35 



N R 

7-' C?. 



The Kj also form a sufficient statistic for estimating the Xj. and hence the and X* 
The maximum likelihood estimators (MLE) for the >.j are given by 

40 



45 



J'* O;. 



and this the MLE for /.* is given by 



-^ J, N N ^ K 

'° X = Z X* X aX ' 
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E{X ) = z a.x. = r 

A 

s };* is also an unbiased estimator of // with variance given by. 



Var(,X ) = 2: -;7-p — = 2:— p 



which can be estimated by 



15 



Var{X ) = I 



I.I N Pa , 
(I 7^)' 



20 



For the above software development/inspection process N=5. Thus 

^1 = -^i2)(^-^i3)(1->^i4)(l 



^5 
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The block of cells C31 :G35 are the upper limits and these are calculated based upon Poisson probability tails. The 
is numbers represent upper bounds for actual DPU counts entered in the defects observed block or cells. The probability 
that one of the actual DPU values is greater than the corresponding upper bound cell, given our historical assumptions, 
IS at most Alpha_U. In other words, there is at most an Alpha_U x 100% chance that an actual number of defects will 
exceed the corresponding upper limit, given true historical assumptions. If several actuals, say n, exceed their corre- 
sponding upper bounds, then there is at most an ( Alpha_U)" x 1 00% chance that the historical assumptions are correct. 
20 To reject the historical assumptions there are two alternatives hypothesis: 

First, the DPU may exceed expected results due to superior process which is finding more DPU. 
Second, the process is not as effective at preventing DPU from occurring in the process, resulting in more defects 
to be found. 

Only experienced engineering judgement can determine the correct conclusion. In any case, the DPU escaping 
25 and product sigma will be tower than the initial goal. A fourth page in Excel Macros contains the original computer 
programs to calculate Poisson probabilities. See appendix A for the Macros. 

The block of cells C41:G45 are lower limits are calculated based upon Poisson probability tails. The numbers 
represent lower bounds for actual DPU counts entered in the defects observed block of cells. The probability that one 
of the actual DPU values is less than the corresponding lower bound, given our historical assumptions, is at most 
30 Alpha_L. To reject the historical assumptions there are two alternative hypothesis: 

First, the DPU may be less than expected due to an inferior inspection process which is finding fewer defects than 
expected. 

Second, the process is more effective than expected at preventing defects from occurring in the process, resulting 
in fewer defects to be found. 
35 Only experienced engineering judgement can determine the correct conclusion. 

The block of cells C51 ;G55 is initialized to the planning DPU spread by stage. As actual DPU data is collected by 
the project, it is entered to overwrite this data. Finally, the data contains only actual data and based upon actual data 
observed, the sigma of escaping DPU is calculated. 

The confidence level E58 represents the probability that the true values of DPU, dmpo, and sigma in the corre- 
40 sponding cells are within the upper and lower bounds indicated. The sigma interval labeled in this program represents 
a confidence interval predicting the sigma for escaping DPU based upon this programs actual data. The interval labeled 
average represents a confidence interval based upon repeating the experiment of collecting actuals multiple times for 
this program. The estimator can be used to generate an approximate confidence interval for the mean number of 
escaping defects, as well as an approximate prediction interval for the actual number of escaping defects. 
45 The first block of cells in the Summary page as illustrated in Fig. 8 reminds the reviewer of the historical stage 

containment data which has been used to model process effectiveness at defect observation. The second block of 
cells represents the data set of actual stage containment DPU for the current project. This data resulted from overwriting 
the planning DPU spread on the current sheet into the defects observed block. The confidence level set on the current 
page is copied to the summary page. It represents the probability that the true values of DPU, dpmo, and sigma in the 
50 corresponding cells are within the upper and lower bounds indicated. The sigma interval labeled in this program rep- 
resents a confidence interval predicting the sigma for escaping DPU based upon this programs actual data. The interval 
labeled average represents a confidence interval for the long-term average DPU level for all projects with the same 
defect generation and inspection characteristics. 

In Fig. 9 there is illustrated a current sheet with equations behind the cells. Cells C81:G86 calculate the total 
55 defects. The Poisson upper limits are cells C31 :G35. The Poisson lower limits are in cells C41 :G45. 

Although the present invention and its advantages have been described in detail, it should be understood that 
various changes, substitutions and alterations can be made herein without departing from the scope of the invention. 
For example, although an embodiment of the invention has been described in which various functions are imple- 
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mented by software^ it will be appreciated that in other embodir^ents, at .east some of those embodiments can be 
implemented in hardware (for example using an ASIC). . 

Claims 

1. A method Of automatically predicting product detects of a future product the method comprisir,g the steps of: 
storing in a first database historical data of defects at different stages of development from a past product and 

LnalfsDresenlmg a. eoond value ol planned a«lecKp.ropp<>itun«.e5 lor said lulure product 
rS.;.'^?" p-o?ee,in9 means said .irs, and seeond values represenled by said firs, and second signals ,o 

-:rn;it:r='?p;:sr-~^ 

2. The method of claim 1 wherein said method is for predicting software defects and said predictive opportunities for 
20 defects are the number of software lines of code. 

3. The method of claim 2 wherein said value representing escaping defects is a sigma value dependent on DPMO. 

4 The method of claim 2 including the steps of changing the number of defects in said second database for the 
differTn staqes^rom predictive defects to actual defects as actual defects are detected . producing a fourth s gnal 
^^p^entg delrs'based on actual and predictive defects,and displaying said defects based on actual and 

predictive results. 

The method of claim 1 wherein said value representing escaping defects is a sigma value dependent on DPIVIO. 

. The method of claim 1 including the steps of changing the number of defects in said second database for the 
dSferTn t s°ages from predictive defects to actual defects as actual defects are detected , producing a fourth s gnal 
r^^sen ng dVfects ba^ on actual and predictive defects.and displaying said defects based on actual and 
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predictive results. 

7. An automated method of software defect prediction of a future project, the method comprising the steps of: 

constructing project phase defect containment da.a from past one or more projects and quality level in terms 

of escaping defects and converting said project containment data to conditional probabilities . 

proS a fts, Signal representing software lines of code and a second signal representing quality level in 

greralTarcHdat^b^ of predictive defects at different stages o, development for said future product 
as a function of said conditional probabilities and said first and second signals: and 
displaying said predictive defects at different stages of development for said future product. 

8 The method of claim 7 including the steps of changing the number of defects in said second database for ^^^^^ 
dmerent stages from predictive defects to actual defects as actual defects are detected , producing a fourth s gnal 
^^esenS defects based on actual and predictive defects.and displaying said defects based on actual and 

predictive results. 

9. A method of predicting software defects in a computer program, the method comprising the steps of: 

automatically generating and displaying on a display means a historical sheet and a current sheet for providing 
data representing defects at different stages of software design: ^, ^ 

entering in said historical sheet data representing defects at different stages of design and s.gma value cor- 
responding thereto based on previous designs: 

entering estimated efficiencies of said historical data: . . , ^ , k^^«h ir. 

responding to said entered data automatically to generate and to display a distribution of defects based ,n 
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efficiencies; 

entering proposed number of lines of code and signal value for current code to be predicted 

backsolving said sigma value and said lines of code to provide a count of total defects and distributing said 

defects based on said percentage of distribution of defects and providing on said current sheet and displaying 

said current sheet: ^' / y 

calculating upper bounds for initial defect spread: 

displaying said upper bounds for observed defect spread; 

calculating lower bounds for initial defect spread; and 

displaying said lower bounds for observed defect spread. 

10. Apparatus for automatically predicting software defects comprising: 

storage means for storing historical data of defects at different stages of software development' 
calculating means coupled to said storing means for calculating a sigma value 

sa^^ stgma TaluT """'^'^"^ ^^"^ ^^'"''"^ "^^^""^ ^""^ calculating means for displaying said historical data and 

means responsive to entry of a new sigma value for backsolving said sigma value to provide predictive defects 
at different stages of software development. 

11. A software defect predictive engine comprising: 

a processor: 

l"Sf!T'' t'n ^i9"3's representing historical data 
on defects at different stages of a software design. 

said processor being responsive to said first electrical signal and estimated prediction efficiencies for providing 

second signals representing the spread of defects at the different stages 

a second computer storage for storing current data relative to software being predicted 

and nroJ;Hri?'"^i ^''^""T """" °' ""'"^^^ ^'^"^^ value entered, to calculate 

and provide e ectrica signals representing total defects projected and being responsive to said second elec- 
tncal signals to calculate and to provide third electrical signals representing the Initial distribution of defec's 

for the current software program: 

said second computer storage storing said third electrical signals and 

for rJni^ ^^^P'^djo said processor and said second computer storage and responsive to said electncal signals 
for displaying said initial distribution of defects of the current software program. a' signals 

12. A software defect predictive engine according to claim 11 ■ wherein- 

currenTsofttarfpr'oVrT"'" '° ^^^"^'^ '^^-^'^^'"^ ^ -^^^^^^'^ °' ^^'-'^ --P-^ - 'he 
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